Embedding location data in video

ABSTRACT

Digital watermarking or steganographic data hiding is used to convey location data in video. A digital watermark or steganographic signal may associate geovector or geolocation information with video, video objects or areas depicted in the video. Other implementations and combinations are provided as well.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.10/371,995 (now U.S. Pat. No. 7,254,249), which claims the benefit ofU.S. Provisional Patent Application No. 60/359,041, filed Feb. 20, 2002.The Ser. No. 10/371,995 application is also a continuation-in-part ofU.S. patent application Ser. No. 10/002,954, filed Oct. 23, 2001 (nowU.S. Pat. No. 7,042,470), which is a continuation-in-part of U.S. patentapplication Ser. No. 09/800,093, filed Mar. 5, 2001 (now U.S. Pat. No.7,061,510). The 10/002,954 application also claims the benefit of U.S.Provisional Application Nos. 60/284,163, filed Apr. 16, 2001,60/284,776, filed Apr. 18, 2001. The present application is also relatedto U.S. patent application Ser. No. 09/833,013 (now U.S. Pat. No.7,249,257). Each of the above patent applications is hereby incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention relates to digitally watermarking video andimages, and is particularly illustrated in the context of video capturedfrom aerial platforms.

BACKGROUND AND SUMMARY OF THE INVENTION

Acquisition of aerial imagery traces its history back to the Wrightbrothers, and is now commonly performed from satellite and space shuttleplatforms, in addition to aircraft.

While the earliest aerial imagery relied on conventional filmtechnology, a variety of electronic sensors are now more commonly used.Some collect image data corresponding to specific visible, UV or IRfrequency spectra (e.g., the MultiSpectral Scanner and Thematic Mapperused by the Landsat satellites). Others use wide band sensors. Stillothers use radar or laser systems (sometimes stereo) to sensetopological features in 3 dimensions. Some satellites can even collectribbon imagery (e.g., a raster-like, 1-dimensional terrestrialrepresentation, which is pieced together with other such adjacentribbons).

The quality of the imagery has also constantly improved. Some satellitesystems are now capable of acquiring image and topological data having aresolution of less than a meter. Aircraft imagery, collected from loweraltitudes, provides still greater resolution.

Such imagery can be used to develop maps or models, such as DigitalElevation Models (DEM) and others. DEM, essentially, is an “elevationmap” of the earth (or part thereof). One popular DEM is maintained bythe U.S. Geological Survey and details terrain elevations at regularlyspaced intervals over most of the U.S. More sophisticated DEM databasesare maintained for more demanding applications, and can consider detailssuch as the earth's pseudo pear shape, in addition to more localizedfeatures. Resolution of sophisticated DEMs can get well below one metercross-wise, and down to centimeters or less in actual elevation.DEMs—with their elevation data—are sometimes supplemented by albedo maps(sometimes termed texture maps, or reflectance maps) that detail, e.g.,a grey scale value for each pixel in the image, conveying aphotographic-like representation of an area.

(There is a large body of patent literature that illustrates DEM systemsand technology. For example: U.S. Pat. No. 5,608,405 details a method ofgenerating a Digital Elevation Model from the interference patternresulting from two co-registered synthetic aperture radar images. U.S.Pat. No. 5,926,581 discloses a technique for generating a DigitalElevation Model from two images of ground terrain, by reference tocommon features in the two images, and registration-mapping functionsthat relate the images to a ground plane reference system. U.S. Pat.Nos. 5,974,423, 6,023,278 and 6,177,943 disclose techniques by which aDigital Elevation Model can be transformed into polygonal models,thereby reducing storage requirements, and facilitating display incertain graphics display systems. U.S. Pat. Nos. 5,995,681 and 5,550,937detail methods for real-time updating of a Digital Elevation Model (or areference image based thereon), and are particularly suited forapplications in which the terrain being mapped is not static but issubject, e.g., to movement or destruction of mapped features. Thedisclosed arrangement iteratively cross-correlates new image data withthe reference image, automatically adjusting the geometry modelassociated with the image sensor, thereby accurately co-registering thenew image relative to the reference image. Areas of discrepancy can bequickly identified, and the DEM/reference image can be updatedaccordingly. U.S. Pat. No. 6,150,972 details how interferometricsynthetic aperture radar data can be used to generate a DigitalElevation Model. Each of these patents is herein incorporated byreference.).

From systems such as the foregoing, and others, a huge quantity ofaerial imagery is constantly being collected. Management andcoordination of the resulting large data sets is a growing problem.Integrating the imagery with related, often adjacent, imagery, andefficiently updating “stale” imagery is also a problem.

In accordance with one aspect of the present invention, digitalwatermarking technology is employed to help track such imagery, and canalso provide audit trail, serialization, anti-copying, and otherbenefits.

In accordance with another aspect of the invention, video imagery isembedded to include location information. The location information mayinclude or point to a so-called “geovector.”

The foregoing and additional features and advantages of the presentinvention will be more readily apparent from the following detaileddescription with reference to the following figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates imagery, which is segmented into image patches.

FIGS. 2 a and 2 b illustrate a correlation of image patches.

FIG. 3 is a flow diagram illustrating an image management methodaccording to one aspect of the present invention.

FIG. 4 is a flow diagram illustrating a method of embedding a geovectorin image data including video; and

FIG. 5 is a flow diagram illustrating a method of decoding an embeddedwatermark to access a database.

FIG. 6 shows a video frame including an object with respect to areference point;

FIG. 7 a shows a plurality of video frames including embedded data; andFIG. 7 b is an image including multiple areas.

DETAILED DESCRIPTION

For expository convenience, the following section focuses on satelliteand aerial “imagery” to illustrate the principles of the invention. Theprinciples of the invention, however, are equally applicable to otherforms of aerial surveillance data and other topographic/mappinginformation. Accordingly, the term “image” should be used to encompassall such other data sets, and the term “pixel” should be construed toencompass component data from such other data sets.

When new aerial imagery is received, it is generally necessary toidentify the precise piece of earth to which it corresponds. Thisoperation, termed “georeferencing” or “geocoding,” can be a convolutedart and science.

In many systems, the georeferencing begins with a master referencesystem (e.g., latitude and longitude) that takes into account theearth's known deformities from a sphere. Onto this reference system theposition of the depicted region is inferred, e.g., by consideration ofthe satellite's position and orientation (ephemeris data), opticalattributes of the satellite's imaging system (e.g., resolution,magnification, etc.), and models of the dispersion/refraction introducedby the earth's atmosphere.

In applications where precise accuracy is required, the foregoing,“ephemeris,” position determination is refined by comparing features inthe image with the placement of known features on the earth's surface(e.g., buildings and other man-placed objects, geological features,etc.) and compensating the georeference determination accordingly. Thus,for example, if the actual latitude and longitude of a building is known(e.g., by measurement from a ground survey—“ground truth”), and thecorresponding latitude and longitude of that building as indicated inthe georeferenced satellite imagery is different, the reference systemapplied to the satellite data can be altered to achieve a match.(Commonly, three or more such ground truth points are used so as toassure accurate correction.)

Ground-truthing is a tedious undertaking. While computer methods can beused to facilitate the process, the best ground truth correction ofimagery generally requires some human involvement. This is impracticalfor many applications.

As shown in FIG. 1, aerial imagery can be segmented into area sets(e.g., image “patches”). These patches can be pieced together (or“composited”) in a quilt-like manner to form a master map. (A “master”map is used generally herein to represent a map or other arearepresentation, typically which will include a plurality of imagepatches. Image patches are defined broadly and may include imagesegments, photographs, separate images, etc.). An image patch mayinclude imagery representing an area, such as a 1×1 meter area, a 1×1kilometer area, etc. Often, an image patch is combined with adjacentpatches, which were gathered on different dates. For example, an imagetaken last week (e.g., Patch C in FIG. 1) may be quilted together withimage patches taken today (e.g., Patch B), or a year ago (e.g., PatchA), to form a larger area map. Also, patches may be replaced over timeto reflect new area developments or movements. (Of course, a master mapneed not be physically pieced together, but may instead beelectronically maintained by a computer database, which correlates thepatches or stores information, e.g., coordinates, patch locations,etc.).

Similarly, image patches can be pieced together with other images takenfrom different aerial platforms (e.g., satellites, airplanes, unmannedaircraft, etc.) or taken with different imagery characteristics.(Imagery characteristics may include resolution, angle, scale, rotation,skew, time, azimuth, device characteristics, altitude, attitude,physical conditions such as cloud cover and magnification, etc.)

Images typically undergo auto-correlation processes to reconciledifferences between adjacent patches, prior to being composited (orarranged) with other patches. A variety of known mathematical techniquescan be utilized in this operation, including dot product computation,transforming to spatial frequency domain, convolution, etc. In a laysense, the correlation can be imagined as sliding one map over the otheror matching pieces in a puzzle-like fashion until the best registrationbetween the two image patches is obtained.

Now consider a geo-referencing example. A new satellite image isacquired corresponding to part of a region represented by a master map.The particular terrain depicted by the satellite image can be inferredfrom ephemeris and other factors, as noted above. By such techniques,the location of the depicted image on the earth's surface (e.g., thelatitude and longitude of a point at the center of the image) may bedetermined within an error of, say 5-500 meters. This is a grossgeo-referencing operation.

Next a fine geo-referencing operation is automatically performed, asfollows. An excerpt of a master map is retrieved from a database—largeenough to encompass the new image and its possible placement error(e.g., an area centered on the same latitude/longitude, but extending250 meters further at each edge). A projective image is formed from thismaster DEM/map excerpt, considering, e.g., the satellite's position andatmospheric effects, thereby simulating how the master map would look tothe satellite, taking into account—where possible—the particular datarepresented by the satellite image, e.g., the frequency bands imaged,etc. (An albedo map may be back-projected on the 3D DEM data in somearrangements to augment the realism of the projective image.).

The projective image formed from the master DEM/map excerpt differssomewhat from the image actually acquired by the satellite. Thisdifference is due, in part, to the error in the gross geo-referencing.(Other differences may arise, e.g., by physical changes in the regiondepicted since the master DEM/map was compiled.).

The projective image is next automatically correlated with the satelliteimage. From the correlation operation, the center-to-center offsetbetween the excerpt of the master DEM/map, and the satellite image, isdetermined. The satellite image can thereby be accurately placed in thecontext of the master map. Depending on system parameters, a fineplacement accuracy of, e.g., between 5 cm and 5 meters (i.e., sub-pixelaccuracy) may be achieved.

(In some embodiments, affine transformations can be applied to the imagedata to further enhance the correlation. E.g., particular geological orother features in the two data sets can be identified, and the satellitedata (e.g., map or image) can then be affine-transformed so that thesefeatures correctly register.).

With the satellite image thus finely geo-referenced to the masterDEM/map, it can be transformed (e.g., resampled) as necessary tocorrespond to the (typically rectilinear) reference system used in themaster map, and then used to refine the data represented in the map.Buildings or other features newly depicted in the satellite image, forexample, can be newly represented in the master map. The master map canbe similarly updated to account for erosion and other topologicalchanges revealed by the new satellite image.

In one embodiment, the finely geo-referenced satellite data is segmentedinto region or area sets, e.g., rectangular patches corresponding toterrain 1000 meters on a side, and each patch is given its own weightingfactor, etc. In a system with 10 meter resolution (i.e., a pixel size of10 m², the patch thus comprises an array of 100×100 pixels. (In someembodiments, the fine geo-referencing is done following the segmentationof the image, with each patch separately correlated with a correspondingarea in the master map.) Each patch may take the form of a separate datafile.

When the new satellite data is added to update the master map, old datamay be discarded so that it no longer influences the map. Consider anarea that is imaged monthly by a satellite. Several months' worth ofimage data may be composited to yield the master map (e.g., so cloudcover that obscured a region in the latest fly-over does not leave partof the map undefined). As each component image data gets older, it maybe given less and less weight, until it no longer forms any part of themaster map. (Other component data, in contrast, may be retained for muchlonger periods of time. Map information collected by ground surveys orother forms of “ground truth” information may fall into this category.).

The master map may be physically maintained in different ways. In oneexemplary arrangement, a database stores the ten sets of data (e.g.,acquired from different sources, or at different times) for each1000×1000 meter patch. When interrogated to produce a map or other data,the database recalls the 10 data sets for each patch, and combines themon the fly according to associated weighting factors and other criteria(e.g., viewing angle) to yield a net representation for that patch. Thiscomposite patch is then combined (e.g., graphically stitched) with otheradjoining, similarly formed composite patches, to yield a data setrepresenting the desired area.

In another embodiment, the component sets of image data are notseparately maintained. Rather, each new set of image data is used toupdate a stored map. If the new image data is of high quality (e.g.,good atmospheric seeing conditions, and acquired with a high resolutionimaging device), then the new data may be combined with the existing mapwith a 20/80 weighting (i.e., the existing map is given a weightfour-times that of the new data). If the new image data is of lowquality, it may be combined with the existing map with a 5/95 weighting.The revised map is then stored, and the new data needn't thereafter betracked.

(The foregoing examples are simplifications, but serve to illustrate arange of approaches.)

The former arrangement—with the component data stored—is preferred formany applications, since the database can be queried to yield differentinformation. For example, the database can be queried to generate asynthesized image of terrain, as it would look at a particular time ofday, imaged in a specified IR frequency band, from a specified vantagepoint.

It will be recognized that a key requirement—especially of the formerarrangement—is a sophisticated data management system. For each data setrepresenting a component 1000×1000 meter patch stored in the database, alarge quantity of ancillary data (meta data) must be tracked. Among thismeta data may be a weighting factor (e.g., based on seeing conditionsand sensor attributes), an acquisition date and time (from which anage-based weighting factor may be determined), the ID of thesensor/satellite that acquired that data, ephemeris data from the timeof acquisition, the frequency band imaged, the geo-referenced positionof the patch (e.g., latitude/longitude), etc., etc. (Much of this datamay be common to all patches from a single image.).

Classically, each component source of data to the system (here referredto as an “image” for expository convenience) is associated with a uniqueidentifier. Tapes and data files, for example, may have headers in whichthis identifier is stored. The header may also include all of the metadata that is to be associated with that file. Or the identifier canidentify a particular database record at which the corresponding metadata is stored. Or hybrid approaches can be used (e.g., the header caninclude a file identifier that identifies a data base record, but alsoincludes data specifying the date/time of data acquisition).

In the final analysis, any form of very reliable image identificationmay suffice for use in such a system. The header approach just discussedis straightforward. Preferable, however, is to embed one or moreidentifiers directly into the image data itself (i.e., “in band”steganographic encoding using digital watermarking). A well-designedwatermarking name-space can in fact become a supra-structure overseveral essentially independent serial numbering systems already in useacross a range of satellite sources. Moreover, rudimentarygeoreferencing information can actually be embedded within the watermarkname-space.

For example, on initial acquisition, an initial digital watermark can beapplied to satellite imagery detailing the ephemeris based grossgeoreferencing. Once the image has been finely georeferenced, theexisting watermark can either be overlaid or overwritten with a newwatermark containing the georeferencing information (e.g., “center lat:N34.432-4352, long: W87.2883134; rot from N/S: 3.232; x2.343, y2.340,dx0.123, dy493, etc.”). These numbers essentially encode georeferencinginformation including projective and atmospheric distortions, such thatwhen this image is corrected, high accuracy should be achieved.

The assignee's U.S. Pat. No. 6,122,403, and pending U.S. patentapplication Ser. No. 09/503,881 (now U.S. Pat. No. 6,614,914), detailsuitable digital watermarking techniques in which values of pixels,e.g., in a 100×100 pixel patch, can be slightly altered so as to conveya plural-bit payload, without impairing use of the pixel data for itsintended purpose. (This patent and patent application are hereinincorporated by reference.). The payload may be on the order of 50-250bits, depending on the particular form of encoding (e.g., convolution,turbo, or BCH coding can be employed to provide some error-correctingcapability), and the number of bits per pixel. Larger payloads can beconveyed through larger image patches. (Larger payloads can also beconveyed by encoding the information is a less robust fashion, or bymaking the encoding more relatively visible.). The watermark payload canconvey an image identifier, and may convey other meta data as well. Insome systems, the component image files are tagged both by digitalwatermark identifiers and also by conventional out-of-band techniques,such as header data, thereby affording data redundancy. Of course, thereare many watermarking techniques known to those skilled in the art, andsuch may be suitably interchanged with the above-cited patent documents.

Watermarking may be performed in stages, at different times. Forexample, an identifier can be watermarked into an image relatively earlyin the process, and other information (such as finely geo-referencedlatitude/longitude) can be watermarked later. A single watermark can beused, with different payload bits written at different times. (Inwatermark systems employing pseudo-random data or noise (PN), e.g., torandomize some aspect of the payload's encoding, the same PN data can beused at both times, with different payload bits encoded at the differenttimes.)

Alternatively, different watermarks can be applied to convey differentdata. The watermarks can be of the same general type (e.g., PN based,but using different PN data). Or different forms of watermark can beused (e.g., one that encodes by adding an overlay signal to arepresentation of the image in the pixel domain, another that encodes byslightly altering DCT coefficients corresponding to the image in aspatial frequency domain, and another that encodes by slightly alteringwavelet coefficients corresponding to the image. Of course, otherwatermarking techniques may be used as suitable replacements for thosediscussed above.).

In some multiple-watermarking approaches, a first watermark is appliedbefore the satellite image is segmented into patches. A later watermarkcan be applied after segmentation. (The former watermark is typicallydesigned so as to be detectable from even small excerpts of the originalimage.)

A watermark can be even applied by the imaging instrument. In someembodiments, the image is acquired through an LCD optical shutter, orother programmable optical device, that imparts an inconspicuouspatterning to the image as it is captured. (One particular opticaltechnique for watermark encoding is detailed in U.S. Pat. No. 5,930,369,which is herein incorporated by reference.). Or the watermarking can beeffected by systems in the satellite (or other aerial platform) thatprocess the acquired data prior to transmission to a ground station. Insome systems, the image data is compressed for transmission—discardinginformation that is not important. The compression algorithm can discardinformation in a manner calculated so that the remaining data is therebyencoded with a watermark.

The ground station receiving the satellite transmission can likewiseapply a watermark to the image data. So can each subsequent systemthrough which the data passes.

As indicated, the watermark(s) can identify the imaging system, thedate/time of data acquisition, satellite ephemeris data, the identity ofintervening systems through which the data passed, etc. One or morewatermarks can stamp the image with unique identifiers used insubsequent management of the image data, or in management of meta dataassociated with the image.

A watermark can also serve a function akin to a hyperlink, e.g., asdetailed in U.S. patent application Ser. No. 09/571,422, which is hereinincorporated by reference. For example, a user terminal can permit anoperator to right-click on a region of interest in a displayed image. Inresponse, the system can respond with a menu of options—one of which isLink Through Watermark(s). If the user selects this option, a watermarkdetection function is invoked that decodes a watermark payload from thedisplayed image (or from a portion of the image in which the operatorclicked). Using data from the decoded watermark payload, the terminalinterrogates a database for a corresponding record. That record canreturn to the terminal certain stored information relating to thedisplayed image. For example, the database can present on the terminalscreen a listing of hyperlinks leading to other images depicting thesame area. By clicking on such a link, the corresponding image isdisplayed. Or the database can present, on the user terminal screen, themeta-data associated with the image.

In some embodiments, watermarks in component images may carry-throughinto the master DEM/map representation. If an excerpt of the masterDEM/map is displayed, the user may invoke the Link Through Watermark(s)function. Corresponding options may be presented. For example, the usermay be given the option of viewing each of the component images/datasets that contributed to the portion of the master map being viewed.

(It will be recognized that a variety of user interface techniques otherthan right-clicking, and selecting from a menu of options therebydisplayed, can be employed. That interface is illustrative only.)

In some embodiments, a watermark can be applied to each DEM/map from themaster database as it is retrieved and output to the user. The watermarkcan indicate (i.e., by direct encoding, or by pointing to a databaserecord) certain data related to the compiled data set, such as thedate/time of creation, the ID of the person who queried the database,the component datasets used in preparing the output data, the databaseused in compiling the output data, etc. Thereafter, if this output datais printed, or stored for later use, the watermark persists, permittingthis information to be later ascertained.

Correlating Images with Watermarks

With reference to FIGS. 2 a and 2 b, watermarks can assist in correctionor correlating imagery characteristics (e.g., such as scale, rotation,resolution, skew, time-matching, etc.). For example, an embeddedwatermark payload may indicate the angle of the imaging device (e.g.,optical camera, imaging sensor, etc.), the height to the imaging device,the relative position (e.g., skew, rotation, etc.) of the device withrespect to a target area, and the resolution of the device and image.(Such measurements can be provided from sensing and positioningequipment on board or in communication with the aerial platform. Suchcharacteristics may be alternatively determined by the georeferencingtechniques discussed above. Of course, other imagery characteristicdetermining techniques may be suitably interchanged with the presentinvention.). Returning to FIGS. 2 a and 2 b, imagery characteristicsprovide information to help manipulate patches A and B (FIG. 2 a) into astandardized or compatible format (FIG. 2 b). Information pertaining tothe imaging characteristics can be used to improve and expedite theauto-correlation processes discussed above. In addition, once theimaging characteristics are known, straightforward processing canmanipulate an image patch to conform to adjacent patches (or to the mapitself). For example, some or all of the patches in a master map aremathematically manipulated to achieve the same scale, orientation,and/or resolution.

With respect to a watermark payload, the imaging characteristics can bedirectly encoded as the watermark payload. Alternatively, an index (oridentifier) associated with a set of these characteristics may beencoded in the payload. To illustrate, a numerical code or indexrepresents a set possible imagery characteristics (or a subset of such).The imagery characteristics are stored in a data base record. The codeor index, once extracted from a watermark, is used to interrogate adatabase to obtain the corresponding data record. (As an alternative, apayload signifies a predetermined set of values, e.g., a payload of 1237signifies a predetermined scale, rotation, skew, etc. Or the indexrelates to a predetermined range of characteristics. For example, therange may specify that the scale is in a particular range, or that theresolution falls within a range, etc.). A watermark payload size andcomplexity can be reduced with a database/index system.

Embedding imagery characteristic in the form of a digital watermarkassists in downstream processing and handling of an image. Anautomated-quilting process can be employed to match patches according tothe georeferencing and/or imagery characteristics provided by a digitalwatermark. These georeferencing and/or imagery characteristics can alsoserve to preserve the historic information about the image and depictedarea. Individual patches can also be watermarked to include coordinatesor master map locators. With reference to FIG. 1, patch E may includecoordinates or a plurality of coordinates that identify its master maplocation, coordinates for corners or edges (e.g., either physicalgeo-coordinates or coordinates relative to its master map location), orits relationship with adjacent patches. Such a locator can be added oncea master map is composited (e.g., by watermarking the master map).Alternatively, such locators can be embedded before quilting, such aswhen imagery is collected or processed.

A time-tag (or stamp) may also be embedded in imagery. The time-tag canbe used to categorize images based on time (e.g., hour, minutes, date,etc.), and to help identify stale or outdated imagery. The time-tag mayoptionally include a plurality of fields, such as time-taken, timeprocessed, time integrating in a master map, etc. Such time-taggingmethods improve management of images. (In one embodiment, an automatedprocess searches a master map database, looking for stale or outdatedpatches, based on predetermined criteria. Once found the stale imagepatch is preferably removed and an updated image patch is inserted inits place.).

FIG. 3 illustrates a flow diagram of an inventive method according toone embodiment of the present invention. Image data is received into asystem or process. The image data is embedded with image characteristics(step S10). Alternatively, the image is embedded with an identifier(index) for database interrogation. The embedded image data is thencorrelated or manipulated to conform to adjacent patches or to maprequirements (step S11). In this regard, the correlation may eitherrender adjacent patches to have approximate (e.g., similar or in a rangeof) imagery characteristics, or to have nearly identical imagerycharacteristics. Or the correlation may group neighboring patches into aset. A map is then generated or constructed (S12). (A map can be quiltedtogether to include many image patches. The digital watermarkidentifiers are used to correlate the image.).

Geo-Locators and Digital Watermarks

Digital watermarking is now disclosed as a central element in a digitalasset management system, particularly for photograph assets (including“digital images”). Copyright labeling, active copyright communications,marketing links, etc., have been explored in the watermark art. Thissection discloses how digital watermarking (and related database linkingproperties) and georeferenced photography inter-relate. In oneembodiment, digital watermarking is used as a platform to simplify andtransform georeferenced photography.

Within the universe of subject matter for photography is what is broadlyreferred to as “remote sensing.” For this discussion, remote sensing isdefined to include all types of photography, which somehow images theEarth's surface or its landscape. Of course, while remote sensing may befacilitated with aerial platforms, such is not required. Add to theremote sensing class, all photography, which somehow has an innateconnection to a location on the Earth—referred herein as “georeferencedphotography.” In the final analysis, virtually all photographs, one wayor another, have innate geographic properties. (Even purely syntheticimages are created by an author located “somewhere.”). Most photographs,including swept-scan satellite imagery and radar, also includingvacation snaps at, e.g., Niagara Falls, can be described as havinginnate, if not explicit, geographic properties. “Time” can also beincluded as an identifying property. (To simplify the discussion, theterms photograph, image, and photography are used interchangeablyhereafter. Such terms may include remote sensing, georeferencedphotography, image, imagery, photos, electronic images or imagery and/oraerial photo, etc., etc.).

Virtually all images can be referenced by a dimensional location vector(e.g., a “geovector”) relative to the Earth's coordinate system. In afirst embodiment, the geovector is presented as a six (6) elementvector, including:

-   -   Latitude;    -   Longitude;    -   Height/Altitude (e.g., as compared to a mean-sea level sphere        with an arbitrary time origin);    -   Time (including date);    -   Cardinal Direction; and    -   Azimuth.

The cardinal direction and azimuth elements can be used to determine aviewpoint depicted in a photograph (e.g., the azimuthal direction of aviewpoint for a given geo-position.). In a modification, cardinaldirection and azimuth indicate the vantage point of the imaging sensor.In still another modification, azimuth and cardinal direction are usedto represent other directional indicators. Of course, the cardinaldirection can be used to orient an image depicted in the photograph.(Although the term “geovector” is introduced in connection with a six(6) dimensional vector, the present invention is not so limited. Indeed,a geovector is defined broadly herein to include information conveyinglocation and/or directional specifying data, regardless of vectorsize.).

In a modification to the first embodiment, a geovector includes “6+1”elements. The extra “+1” dimension can be multi-dimensional in nature,generally representing “sensor geometry.” Sensor geometry is definedbroadly herein to include a coherent set of optical (or electrical)sampling functions, e.g., corresponding to each pixel (or pixel block)and/or a microdensity region of a photograph. Of course, there is avariety of other types of sensor geometry, each associated with variousrules on how the geometry is defined and how it affects the referencingparameters (e.g., latitude, longitude, height, etc.). A common form ofsensor geometry is a rectangular fan or pyramid centered on a camera'saperture, which can be used as a stand-in for many others forms. Ofcourse, there are many other geometry forms known to one of ordinaryskill in the art, which are suitably interchangeable with the presentinvention.

The march of technological progress is transitioning more photographyfrom the “innate” category to the “explicit” category through the use ofglobal positioning system (GPS) technology and/or local wirelesstechnologies. GPS can be used to determine a physical location (e.g.,including properties of a geovector). As will be appreciated by thoseskilled in the art, GPS is a satellite-based radio navigation systemcapable of providing continuous position, velocity, and timeinformation. GPS receiver units receive positioning signals from aconstellation of satellites deployed in various orbits about earth(e.g., 12-hour orbits). The satellites continuously emit electronic GPSsignals (or telemetry) for reception by ground, airborne, or watercraftreceiver units. By receiving GPS signals from a plurality of satellites,a properly configured receiver unit can accurately determine itsposition in at least three dimensions (e.g., longitude, latitude, andaltitude/height). Some GPS systems also provide compass-likefunctionality, in which cardinal direction and azimuth are determined.(Alternative methods can be used to determine a geovector. For example,many terrestrial-based stations emit navigational beacons that can beused to determine geo-location and relational-direction. Wirelesssystems may also be used to triangulate signals emitted from a mobiledevice. Such signals are collected at multiple receiving locations andbased on the relative reception time and/or strength a geo-location isdetermined for the mobile device. Similarly, a mobile device cantriangulate its position based on received beacons.).

The georeferencing techniques discussed above and in the incorporated byreference patents and applications can also be used to determinegeovector information corresponding to a location depicted in aphotograph. (E.g., a GPS or wireless system can provide geovectorinformation. Or geovector information can be obtained from an imagecapture device, among the other techniques discussed.). In oneembodiment, geovector data is obtained via an online (e.g., internet ornetwork) database. A user simply enters in a street address or map-gridlocation, and the database returns corresponding geovector data (e.g.,longitude, latitude, height, etc.). Or the geovector information isobtained from a user or agency based on human or computer analysis of animage. Artisans know other ways to determine and obtain geovectorinformation. Of course, such other known techniques are suitablyinterchangeable with the present invention.

Beginning with the area of remote sensing, and extending to allphotography with an innate geovector, digital watermarking is extendedto embrace this fundamental set of information inherent in each andevery photograph. Just as a “copyright” is fundamentally a part of everyphotograph, so too is a “geovector” a fundamental part of everyphotograph, and digital watermarking can expressly convey this geovectorinformation.

Once obtained, a geovector is either contained in the embedded watermarkinformation itself, or contained in a database to which the watermarkrepresents a pointer, or both (see FIG. 4). Indeed, the geovector can beincluded in a watermark message or payload. In one embodiment, awatermark embedder performs error correction coding of a watermark'sbinary message (e.g., representing the geovector), and then combines thebinary message with a carrier signal to create a component of awatermark signal. There are several error correction coding schemes thatmay be employed. Some examples include BCH, convolution, Reed Solomon,and turbo codes. These forms of error correction coding are sometimesused in communication applications where data is encoded in a carriersignal that transfers the encoded data from one place to another. In thedigital watermarking application discussed here, raw bit data can beencoded in a fundamental carrier signal. It then combines the watermarksignal with a host signal (e.g., a photograph). Further discussion forembedding messages can be found in assignee's U.S. patent applicationSer. No. 09/503,881 (now U.S. Pat. No. 6,614,914), mentioned above.Artisans know other embedding techniques that are suitablyinterchangeable with the present invention.

A watermark embedded within a photograph may serve as (or carry) adatabase index or pointer. For example, the watermark includes an index,which once decoded, is used to interrogate a database (see FIG. 5). Thedatabase preferably contains data records including geovectorinformation. The watermark index is used to identify a correspondingdata record for the respective photograph (e.g., the photograph in whichthe watermark is embedded within). Of course, the database may be localor may be remotely accessed. In one embodiment, the watermark includesdata corresponding to a URL or IP address, which is used to access awebsite. See Assignee's U.S. patent application Ser. No. 09/571,422 (nowU.S. Pat. No. 6,947,571), mentioned above, for a further discussion ofwatermark-based linking. (The data may directly include the URL or maybe used to access the URL.). A database associated with the website maybe interrogated to retrieve the corresponding geovector information fora photograph. (In another embodiment, a watermarking reading devicedefaults to a URL or to an IP address, or queries a default database,upon detection of a watermark in a photograph.).

In yet another embodiment, geovector information is redundantly providedin header structures and watermark payloads.

Standardization efforts are currently underway, which are extending theidea of the geovector well beyond the examples presented above. See, forexample, the Open GIS Consortium, an international consortium seeking tofoster collaborative development of the OpenGIS Specifications tosupport full integration of geospatial data and geoprocessing resourcesinto mainstream computing (http://www.opengis.org). (Of course, thereare other known groups and companies focusing on geospatial andgeographic information and services efforts. The “digital earth” conceptis also known.). Such proposed standards have straightforward coordinatesystems at their core.

We have determined that the standardization proposals lend themselves toconveying georeferencing in several different formats, includingconveying information with digital watermarks, classic headerstructures, and pointer-to-elements in an associated database. Uponcloser examination, however, we believe that our inventive digitalwatermarking techniques provide enhanced benefits when compared to theseother techniques.

In today's world, where photography is rapidly becoming digital, amethod of securely attaching identifying information (e.g., geovectorinformation) to a corresponding photograph is needed. Digitallywatermarking photographs provides a solution for the attachment problem.As discussed, a watermark may provide geovector information (or accessto such information). A photograph many even be redundantly embeddedwith multiple copies of a watermark, further ensuring robust attachmentof information to the photograph. Contrast our digital watermarkingprocedure with a procedure, which appends geovector information viaheaders. Whereas headers may be able to provide geovector information,they have a higher chance of separation from the underlyingdata—defeating a secure attachment feature.

Some of our inventive digital watermarking techniques involve a step ofidentifying a photograph (e.g., digitally watermarking a photograph witha binary identifier or a geovector) and, if using an index oridentifier, storing information related to the index in some database oracross a group of distributed databases. Adding a dimension of geovectorinformation to the management of photographs results in a database orset of coordinated databases, which represent a searchable platformsuitable for geographically based queries. The implications of such aretremendous. For example, a fisherman may search the database(s) for aphotograph of a favorite fishing hole in Wyoming, based on a searchcriteria for a given time period, a range of time periods or bygeo-location. The applications are endless—expanding far beyonddispelling fish stories. Friends of the fisherman may decode a watermarkgeovector or index embedded within the fisherman's watermarkedphotographs (e.g., by a compliant watermark reading device) to determinewhether an area depicted in a photograph corresponds to a trout farm orto a high mountain lake—allowing “fish stories” to be verified. Thisinformation is readily available via a geovector associated with theimage. The fisherman can maintain a photojournal of his fishing trips,knowing that the embedded watermarks provide sufficient information tohelp retrace his steps and travels. To aid this endeavor, digitalcameras are envisioned to be equipped with watermark embedding softwareand geovector gathering modules such as GPS units. Or the geovectorinformation can be added when images are stored in a database orprocessed after the fishing excursion.

Digitally watermarking photographs helps to provide a collision-freeserial numbering system for identifying imagery, owners, and attributes.

There are additional benefits in creating a georeferenced system ofimages using digital watermarks. A classic notion in moststandardizations across all industries is a notion of a “stamp” or“seal” or a similar concept of indicating that some object hassuccessfully completed its appointed rounds. Call it branding, call itformality, or call it a soft form of “authenticity;” the historicalmomentum behind such a branding concept is huge. In one embodiment, toensure that a given image is properly georeferenced (under a chosenstandard), digitally watermarking the given image is a final steprepresenting a formalized “seal of approval.” The digital watermarkitself becomes the seal. In one embodiment, a watermark identifier isobtained from an online repository, which issues and tracks authenticidentifiers. The repository can be queried to determine the date andtime of issue. Or the identifier can be linked to a seal or companylogo. Software and/or hardware is configured to routinely read embeddeddigital watermarks and display an appropriate brand logo, seal, orcertification. The “seal” itself then becomes a functional element of astandardization process, serving many functions including permanentattachment to standardized and dynamic metadata (e.g., a geovector).

Photographs by their very nature can be inter-processed, merged, split,cut up, etc., and so forth, as described in the prior art. This tendencyis especially applicable to various geo-referenced imagery applicationswhere some data sets are merged and viewed as derivative images. (Seeassignee's U.S. patent application Ser. No. 09/858,336, titled “ImageManagement System and Methods Using Digital Watermarks,” filed May 15,2001 (now U.S. Pat. No. 7,098,931), which is herein incorporated byreference.). Tracking image pieces is a daunting task. We have foundthat digital watermarks, in many such applications, are a good way ofcoordinating and keeping track of highly diverse image components. Forexample, an image is redundantly embedded with multiple copies of awatermark including a geovector for the image. When the image is cut up(or merged, etc.), each image piece preferably retains at least one ofthe redundantly embedded watermarks. The watermark is then decoded toidentify the respective image piece, even when the piece is merged orcombined with other image pieces.

A geovector may also provide sufficient information for stitchingtogether map quilts, as discussed above, particularly if boundary orcorner coordinates are provided. Instead of focusing on imagerycharacteristics, the map is quilted together based on the embeddedgeovector information.

The present invention includes many applications beyond identifying andassociating data with photographs. Now consider embedding a digitalwatermark in a particular region of a map or photograph (e.g.,corresponding to a location for a fire hydrant, tree, road, building,lake, stream, forest, manhole, water or gas line, park bench,geographical area, stadium, hospital, school, fence line, boarder,depot, church, store, airport, etc., etc.). These region-specificwatermarks preferably include unique watermark payloads. A watermarkpayload conveys geovector information (or map coordinates) correspondingto its particular region of interest. (E.g., a geovector correspondingto a fire hydrant reveals the hydrant's location in latitude/longitude,etc. coordinates.). Now consider a modification in which, instead ofuniquely watermarking individual map or photograph regions, a digitalwatermark is redundantly embedded throughout the map or photograph. Inthis modification, geovector information is conveyed via the redundantwatermark payload for all fire hydrants's depicted on the map orphotograph. Alternatively, instead of a payload conveying such geovectorinformation, the payload comprises an index, which is used tointerrogate a database to retrieve geovector information. (It should beappreciated that a fire hydrant is used for explanatory purposes only.Of course, other regions, structures, roads, land areas, bodies ofwater, buildings, etc. can be similarly watermarked.).

In another embodiment, a utility company watermarks a map or photographto include geovector information corresponding to specific depictedobjects, such as power stations, transformers and even transmissionlines. Such information assists in locating areas for repair orinspection. Additional information can be stored in a database accordingto its geovector. For example, a power line's capacity, age, maintenancerecord, or rating can be associated in a database according to theline's geovector. Commonly assigned U.S. patent application Ser. No.09/833,013, titled “Digitally Watermarked Maps and Signs and RelatedNavigational Tools,” filed Apr. 10, 2001 (now U.S. Pat. No. 7,249,257),herein incorporated by reference, discloses various techniques forwatermarking and reading maps. Such principles can be applied here aswell. In another embodiment, a city, municipal, state or governmentagency digitally watermarks geovector location information on its mapsand charts, corresponding to streets, country areas, buildings,manholes, airports, ports, water systems, parks, etc.

In another embodiment, school age children carry bracelets, book bags,tags, ID cards, shoelaces, or necklaces, etc., each watermarked withgeovector information identifying their home, parents work address orschool location. When lost, the preschooler presents her bracelet (orother object) to a police officer, school official, or automated kiosk.The embedded watermark is decoded to reveal the geovector information.The child's home or school, or a map route, can be identified from such.

Tags or collars for domestic animals or livestock can be geo-watermarkedto assist in recovery when lost.

In still another embodiment, documents are embedded with geovectorinformation. Consider embedding geovector data on a deed or propertylisting. Additional information regarding the property (e.g., titlehistory, tax information, community information, recording information,photographs, etc.) is obtained via the geovector data link. For example,the additional information can be stored in (or referenced by) adatabase. The geovector data or other pointer serves as the index forthe database.

Geovector information can also assist in notarizing (or authenticating)a document. Data is embedded in the document, which may indicate thedocument time (e.g., date and time) and location of creation (orexecution). Upon presentment to a compliant watermark-reading device,the embedded data is extracted and read for verification.

In yet another embodiment, GPS or geovector information is embedded invideo content. A watermark is embedded in each frame or every n frames.(Or a watermark can be embedded in a random or pseudorandom manner.Still further, a watermark can be embedded over multiple frames, so thatsuccessful watermark decoding requires analysis of the multipleframes.). Assignee's U.S. Pat. No. 6,122,403, and pending U.S. patentapplication Ser. Nos. 09/503,881 (now U.S. Pat. No. 6,614,914),09/342,675 (now U.S. Pat. No. 6,400,827), 09/516,302 (now U.S. Pat. No.6,449,379) and 09/872,199 (now U.S. Pat. No. 6,675,146) disclosesuitable embedding techniques. Of course, other video watermarkembedding techniques may be suitably interchanged with this aspect ofthe present invention so long as the alternative watermarks includesufficient data carrying capacity. For example, the Philip's Watercast™system may be configured to embed GPS or geovector information. In oneinventive implementation, a watermark is embedded to correspond with avideo object (e.g., a depicted building, mountain, road, Coke can, etc.,etc.). Assignee's U.S. patent application Ser. No. 09/597,209 (now U.S.Pat. No. 6,411,725) discloses suitable techniques for such object basedembedding. Regardless of the embedding technique, the digital watermarkpreferably includes a payload or other information bits to carry orotherwise convey GPS or geovector information. Sometimes the payloadincludes a pointer which is used to link to GPS or geovectorinformation.

In one implementation of this video embodiment, a video camera capturesvideo. The video camera can be located in a warehouse, home, spaceshuttle or station, office, aircraft (manned or unmanned), factory,park, satellite, stadium, etc., etc. Or the camera can be hand carriedor mounted to say a car or helicopter. The video camera preferablyincludes or communicates with a GPS receiver. The GPS receiverpreferably provides current location data to a digital watermarkembedder. The embedder is integrated with or otherwise communicates withthe video camera. The embedder embeds the location data in correspondingvideo frames as they are captured by the video camera. (In analternative implementation, a video stream and GPS location data arecommunicated to a remote computer. The remote computer includes adigital watermark embedder to embed the video stream with the locationdata.). The GPS preferably provides updated location data to reflectlocation changes of the video camera and/or the camera's viewingperspective. We note that the video camera and/or the embedder can besynchronized to help ensure that the location data corresponds to theappropriate video frame or frames. Captured video is thus embedded withGPS information across time, e.g., the embedded GPS information changesas the camera/GPS location (or viewing area) changes over time. In apreferred implementation, geovector data is embedded so as to conveylocation information for areas depicted within a video frame. Asgeocoordinates change across a video frame, so also a watermark payloadvaries across the frame in a respective manner.

In one aerial implementation of this video embodiment, location dataembedded in a video frame corresponds to the frame center (or to anotherpredetermined location, e.g., frame edge, or off-center frame location).For example, if the video platform includes an unmanned aircraft, theembedder may embed longitude and latitude of a location depicted in avideo frame's center. A GPS receiver may also provide an altitude of theunmanned aircraft, which can be embedded in the frame, or used todetermine the appropriate ground coordinates for the ground areadepicted in the video or video frame. (We note that the altitude of thevideo capture platform is helpful in resolving off-center coordinates ofother depicted video frame locations.).

In another implementation, embedded location information variesthroughout a video frame. For example, an image frame is broken intoblocks or shapes (e.g., polygon, circles, objects defined by edges, apixel or group of pixels, etc.), and for each block or shape (or groupof blocks or shapes) a watermark includes a unique identifier for thatrespective block or shape. In some implementations, the block or shapeidentifier includes geo-metadata such as ground coordinates, relativegeo-position with respect to a fixed geo-position (e.g., a position orobject depicted in the video frame is 17 kilometers North by Northwestof a fixed location), latitude and longitude, or a relative indicator ofwhere the block or shape falls or is positioned within the video frame.

For a relative frame indicator, the watermark identifier for an objector shape may include an XY spatial position (or other coordinate unit)relative to the frame itself or to a fixed point(s) within the frame.With reference to FIG. 6, object M includes a watermark embedded thereinincluding such an indictor, e.g., a value representing a distance as anamount of pixels from a fixed point FP to the center of the object M.The indicator could additional convey boundary or edge information forobject M relative to the center of the object or relative to the fixedpoint FP. The ground coordinates associated with object M can bedetermined when the ground coordinates of the fixed point FP are knownand the distance/per pixel (or other image/frame information) for theframe is known.

In still another implementation, an automated controller monitors a GPSunit (or mapping model) to determine when captured video corresponds toa predetermined location. The controller activates a watermark embedderto embedded a watermark within the video once the captured videocorresponds with the predetermined area. The watermark includes anidentifier to indicate that the video corresponds with the predeterminedarea. (The presence of the watermark identifier signals thepredetermined location without the need of geo-coordinates.) Thisimplementation is useful to reduce searching time to find a specificgeographic area, since searching for the digital watermark readilyidentifies the predetermined location within a video segment. Toillustrate, consider a Cessna aircraft that takes off from a privateairstrip in French Guiana. The Cessna's mission is to videotape a fireburning in a predetermined location in a rain forest. The Cessnaincludes a video camera, controller, digital watermark embedder and GPSunit. The controller receives signals from the GPS unit and, when theGPS data indicates that the Cessna has reached the predeterminedlocation (e.g., the fire line), actives the watermark embedder. Thewatermark embedder embeds a digital watermark at a specific video framein the captured video. The watermark embedder can continue to embed theidentifier so long as the plane or video captured from the video cameracorresponds with the predetermined location. The video is examined afterthe Cessna returns to the airstrip. The presence of a watermarkidentifier through the sequence of video frames can be used to identifythe predetermined location; the absence of the watermark identifier canbe used to signal that the unmarked video does not depicted thepredetermined location. Or, a second watermark (or different watermarkcomponent) can be used to signal the end of the predetermined location.Of course, multiple watermarks (e.g., with different identifiers) can beused within a video sequence to identify a plurality of differentpredetermined locations.

(In a related implementation dealing with images, a watermark isembedded in an image or image area to indicate that a depicted areacorresponds to a predetermined area. A plurality of images can besearched (e.g., analyzed for watermarks embedded therein) to find theexpected watermark. Once found, the corresponding image is deemed todepict the predetermined location. In an audio implementation, a digitalwatermark is embedded in audio to correspond with audio captured at aspecific location. The watermark may optionally include location datasuch as a geovector.).

In still another implementation, a watermark is used as an odometer ofsorts. A watermark is inserted throughout a video sequence. Thewatermark varies (e.g., increments) in time across captured frames. (Forexample, the watermark payload or identifier varies with time.) Theidentifier preferably corresponds with airspeed of the aerial platformor relates a function of airspeed, altitude of aerial platform and/orflight plan, etc. The watermark identifier can be used to determine aposition of a depicted area in video frame relative to a starting pointor flight path.

A further embodiment is disclosed with reference to FIG. 7 a. Aplurality of video frames (1, 2, 3) each include a region K1, K2 and K3,respectively. Video frames 1, 2 and 3 were captured from a moving aerialplatform, and need not be adjacently arranged in a video sequence.Regions K1, K2 and K3 each depict a different ground location. (If theaerial platform is moving slowing, the difference between these regionsmay be subtle.) The regions occupy the same spatial frame positionrelative to one another as shown. Region K1 includes a first digitalwatermark W1 embedded therein. The first watermark W1 includes a firstpayload P1 that provides a base indication (e.g., coordinates) for theground area depicted in K1 (perhaps a center location or boundary,etc.). In some implementations the indication is an identifier that canbe used to access a data record including the ground coordinates. RegionK2 includes a second digital watermark W2 embedded therein. The secondwatermark W2 includes a second payload P2. Instead of providing theactual coordinates for the depicted area, however, payload P2 includes arepresentation of the change (Δ₁) in ground coordinates between K1 andK2. For example, the change may be represented in terms of geovectordifferences, miles, in latitude/longitude differences, in terms ofpredetermined map coordinates, etc. Similarly, region K3 includes apayload P3 which represents a change (Δ₂) in geo-location between thearea depicted in K3 and the area depicted in region K2 (or between K1).Of course, the respective ground coordinates (and changes incoordinates) can be determined in real-time by a watermark embedder incommunication with a GPS and/or aerial platform information (e.g.,altitude, angle of video capture, flight speed, etc.). Or the embeddingmay occur later, perhaps after some human analysis of the capturedvideo.

A similar technique can be used within a video frame or image, as shownin FIG. 7 b. Here, regions K1, K2 and K3 represent separate (perhapseven overlapping) areas or blocks within an image. Each region includesa watermark embedded therein. The watermarks include payloads. PayloadP1 includes a base indication for the ground coordinates associated withthe area depicted in region K1. As discussed above, payload P1 canphysically carry this information or can provide a link to a data recordincluding the ground coordinates. Region K2 includes a second digitalwatermark having a payload P2. Instead of conveying the actual groundcoordinates for the area depicted in region K2, the payload P2preferably conveys a change of the ground coordinates for the areadepicted in region K2 in comparison to the ground coordinates depictedin K1. Similarly, a watermark payload P3 embedded in region K3 includesa change or difference in ground coordinates with respect to the groundcoordinates of the area depicted in region K2 (or K1).

Of course, other geovector information (e.g., time/date, cardinaldirection, sensor geometry, and/or azimuth, etc.) can be embedded invideo as well, along with camera characteristics such as resolution,optics characteristics, etc. Also, as with other content discussedabove, a geovector can include or be replaced with a pointer. Thepointer can be used to access a database to retrieve the geovectorcoordinates. Also, the geovector can be used as a backbone (or as apersistent identifier) in a video management system. Video segments orvideo frames are indexed in a database according to their geo-location.

In yet another embodiment, geovector information is the common factor,which binds information together. For example, information is storedaccording to its geovector information (e.g., according to creationgeo-location, subject matter geo-location, ancillary relationship to ageo-location, etc.). Database searching for information is carried outvia the geovector data. To illustrate, the database is searched for allinformation pertaining to a specific geo-vector (e.g., the WashingtonMonument). All data (or a subset of the data) pertaining to thegeovector (e.g., the Washington Monument) is returned to the user. Thedata can include reports, web pages, maps, video and audio clips,pictures, statistical data, tourist information, other data, musings,related sonnets, governments information, just to name a few.

These are just a few embodiments and examples employing digitalwatermarking of geovector data. There are many other applications, whichfall within the scope of the present invention.

CONCLUSION

The foregoing are just exemplary implementations of the presentinvention. It will be recognized that there are a great number ofvariations on these basic themes. The foregoing illustrates but a fewapplications of the detailed technology. There are many others.

For example, digital watermarks can be applied to any data set (e.g., asatellite image, video or a map generated from the master database) forforensic tracking purposes. This is particularly useful where severalcopies of the same data set are distributed through different channels(e.g., provided to different users). Each can be “serialized” with adifferent identifier, and a record can be kept of which numbered dataset was provided to which distribution channel. Thereafter, if one ofthe data sets appears in an unexpected context, it can be tracked backto the distribution channel from which it originated.

Some watermarks used in the foregoing embodiments can be “fragile.” Thatis, they can be designed to be lost, or to degrade predictably, when thedata set into which it is embedded is processed in some manner. Thus,for example, a fragile watermark may be designed so that if an image isJPEG compressed and then decompressed, the watermark is lost. Or if theimage is printed, and subsequently scanned back into digital form, thewatermark is corrupted in a foreseeable way. (Fragile watermarktechnology is disclosed, e.g., in application Ser. Nos. 09/234,780,09/433,104, 09/498,223, 60/198,138, 09/562,516, 09/567,405, 09/625,577,09/645,779, and 60/232,163. Each of these patent applications is hereinincorporated by reference.) By such arrangements it is possible to inferhow a data set has been processed by the attributes of a fragilewatermark embedded in the original data set.

Certain “watermark removal” tools can be built to alleviate visibilityor processing problems in cases where unacceptable impact of a digitalwatermark is identified. This can either be a generic tool or one highlyspecialized to the particular application at hand (perhaps employingsecret data associated with that application). In another embodiment, a“remove watermark before analyzing this scene” function is includedwithin analysis software such that 99% of image analysts wouldn't knowor care about the watermarking on/off/on/off functionality as a functionof use/transport.

As will be apparent, the technology detailed herein may be employed inreconnaissance and remote sensing systems, as well as in applicationssuch as guidance of piloted or remotely piloted vehicles. Onceidentified from a map or photograph, geovector data can be uploaded tosuch vehicles.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate by reference, in their entireties,the disclosures of the above-cited patents and applications. Theparticular combinations of elements and features in the above-detailedembodiments are exemplary only; the interchanging and substitution ofthese teachings with other teachings in this application and theincorporated-by-reference patents/applications are expresslycontemplated.

It should be understood that the technology detailed herein can beapplied in the applications detailed in the cited DEM patents, as wellas in other mapping and image, audio or video contexts as well.(Likewise, the technologies detailed in the cited patents can beadvantageously used in embodiments according to the present invention.)

While a geovector is described above to include, e.g., “6+1” dimensions,the present invention is not so limited. Indeed, a geovector can includemore or less vector elements, depending on the referencing precisionrequired. (To illustrate, altitude may be immaterial when othergeovector coordinates are provided. Or a camera sensor geometry (e.g.,“+1”) element may not be needed to uniquely identify a location or toaccount for sensor geometry. Alternatively, a map identifier or locatorcan be included to achieve similar functionality instead of a geovector.In other cases, where only rough referencing information is needed,providing only longitude and latitude coordinates may be sufficient. Ofcourse, in the event that geospatial or geography information andservices standards are formalized and/or updated, the geovector can beformatted to include the reference locators described in that standard.Similarly, instead of a geovector, geo-coordinates or other locationinformation can be provided via a watermark or watermark index. The term“geovector” is sometimes broadly used herein to encompass thesealternatives.).

There are many embodiments discussed herein which may benefit from theinclusion of two different watermarks. For example, a first watermarkmay include information regarding (or pointing to) geovectorinformation, while a second watermark includes a database identifier orlocation. The second watermark may alternatively include (or pointtoward) information pertaining to events, people or animals identifiedin the photograph, occasions, groups, institutions, copyright ownership,etc. Or the embodiment may include both a robust geovector watermark anda copy-tamper fragile watermark.

While particular reference was made to Digital Elevation Models andalbedo maps, the same principles are likewise applicable to other formsof maps, e.g., vegetative, population, area, thermal, etc., etc.

While one of the illustrated embodiments correlated incoming imagerywith a projective image based on the master DEM/map, in otherembodiments a reference other than the master DEM/map may be used. Forexample, a projection based just on part of the historical data fromwhich the DEM/map was compiled can be used (e.g., one or more componentdata sets that are regarded as having the highest accuracy, such asbased directly on ground truths).

Although not belabored, artisans will understand that the systemsdescribed above can be implemented using a variety of hardware andsoftware systems. One embodiment employs a computer or workstation witha large disk library, and capable database software (such as isavailable from Microsoft, Oracle, etc.). The registration, watermarking,and other operations can be performed in accordance with softwareinstructions stored in the disk library or on other storage media, andexecuted by a processor in the computer as needed. (Alternatively,dedicated hardware, or programmable logic circuits, can be employed forsuch operations.).

Certain of the techniques detailed above find far application beyond thecontext in which they are illustrated. For example, equipping an imaginginstrument with an optical shutter that imparts a watermark to an imagefinds application in digital cinema (e.g., in watermarking a theatricalmovie with information indicating the theatre's geo-location, date,time, and/or auditorium location for a given screening).

The various section headings in this application are provided for thereader's convenience and provide no substantive limitations. Thefeatures found in one section may be readily combined with thosefeatures in another section.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of the invention. Rather, we claim as our inventionall such modifications as may come within the scope and spirit of thefollowing claims and equivalents thereof.

1. A method comprising: obtaining geocoordinates representing severaldifferent geo-locations; and embedding the geocoordinates in videodepicting the several different geo-locations with digital watermarking,wherein each of the digital watermarks associated with eachgeocoordinate is spatially aligned within the video so as to be within aregion of a video frame that corresponds to the respective geo-location.2. A method comprising: selecting a geo-location; recording video ofdifferent geo-locations including the selected geo-location, wherein thevideo is recorded by an airborne platform; monitoring a currentgeo-location of the airborne platform; and initiating embedding ofdigital watermarking in the video when the current geo-locationcorresponds to the selected geo-location, wherein the presence of thedigital watermarking without reference to content of the digitalwatermarking at a particular location in the video identifies theselected geo-location.
 3. A method of digital watermarking video, themethod comprising: steganographically hiding a first, plural-bit payloadin data representing a first video portion, wherein the first,plural-bit payload is associated with at least one or more groundcoordinates associated with the first video portion; andsteganographically hiding a second, plural-bit payload in datarepresenting a second video portion, wherein the second, plural-bitpayload is associated with a difference between one or more groundcoordinates associated with the second video portion and the one or moreground coordinates associated with the first video portion.
 4. A methodto identify video, wherein the video comprises a first video framecaptured from an aerial platform, the method comprising:steganographically embedding data in the first video frame, wherein thedata comprises first location data corresponding to at least a firstlocation depicted in the first video frame and second location datacorresponding to at least a second location depicted in the first videoframe, wherein the first location data differs from the second locationdata, wherein the steganographic embedding comprises alterations to thevideo that are generally imperceptible to a human viewer of the video,and wherein the second location data comprises finely referencedlocation data relative to the first location data.
 5. The method ofclaim 4, wherein the first location data and the second location dataeach comprise at least longitude and latitude coordinates.
 6. The methodof claim 4, wherein the first location data and the second location dataeach further comprise time data and at least one of altitude of theaerial platform at video capture or video capture angle.
 7. The methodof claim 4, wherein the first location data and the second location datarespectively correspond to at least one of a first location of theaerial platform or a first aerial platform viewing perspective, and asecond location of the aerial platform or a second aerial platformviewing perspective.
 8. The method of claim 2, further comprisinginitiating embedding of different digital watermarking in the videocaptured by an airborne platform to indicate an end of the selectedgeo-location.
 9. The method of claim 2, wherein the digital watermarkingis limited to portions of the video which correspond to the selectedgeo-location.
 10. The method of claim 9, wherein an absence of thedigital watermarking in some portions of the video indicates that suchportions of the video does not depict the selected geo-location.
 11. Themethod of claim 2, further comprising selecting a second geo-location;initiating embedding of a second digital watermarking in the video whenthe current geo-location corresponds to the selected secondgeo-location, wherein the presence of the second digital watermarking ata particular location in the video identifies the selected secondgeo-location, and wherein the selected second geo-location is identifiedwithin the video without reference to content of the digital watermark.12. The method of claim 1, wherein at least two geocoordinates areembedded in different regions of a video frame.
 13. The method of claim1, wherein the geo-locations comprise one or more locations ofbuildings, roads, and utility company equipment.
 14. The method of claim4, wherein the aerial platform comprises at least one of a satellite,helicopter, unmanned aircraft, space shuttle, or aircraft.